Search Results: "Tollef Fog Heen"

17 December 2008

Tollef Fog Heen: varnishlog's poor man's filtering language

Currently, varnishlog does not support very advanced filtering. If you run it with -o, you can also do a regular expression match on tag While in Brazil, I needed something a bit more expressive. I needed something that would tell me if I had vcl_recv call pass and the URL ended in .jpg.
varnishlog -o -c   perl -ne 'BEGIN   $/ = "";  print if
(/RxURL.*jpg$/m and /VCL_call.*recv pass/);'
fixed this for me.

15 December 2008

Tollef Fog Heen: Ruby/Gems packaging (it's java all over again)

It is sad to see how people complain about how packaging Ruby gems is painful. It seems like it is the Java packaging game all over again where any application ships its dependencies in a lib/ directory (or in the case of Rails, vendor/). Mac OS X applications seem to do some of the same thing by shipping lots of libraries in their application bundle, which is really just a directory with some magic files in it. This is of course just like static linking, which we made away with for most software many years ago, mostly due to the pain associated with any kind of security updates. Update: What I find sad is that people keep making the same mistakes we made and corrected years ago, not that people are complaining about those mistakes.

28 November 2008

Tollef Fog Heen: !Internet

qurzaw (0.0.0.0)                                                 Fri Nov 28 21:34:28 2008
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                             Last  60 pings
 1. 10.125.123.1             ............................................................
 2. 10.84.0.1                .??????.......??????.......??????......???????......??????..
 3. c9110002.virtua.com.br   .????????.....??????..?..?????????..??????????.?..?.??????.?
 4. embratel-G2-0-1-ngacc01. .??????.......??????.......??????......???????......??????..
 5. ebt-T0-5-5-0-21-tcore01. .??????.......??????.......??????......??????.......??????..
 6. 200.230.251.133          .??????.......??????.......??????......??????.......??????.?
 7. 200.230.251.154          .??????.......??????....?.???????......??????.......??????..
 8. ebt-G4-2-intl03.rjo.embr .??????.......??????......???????......??????.......??????..
 9. ebt-ge-5-2-0-intl02.mian .??????......???????......???????......??????.......??????..
10. p4-1-0-3.r01.miamfl02.us .??????......???????...?..???????......??????.?.....??????..
11. xe-1-3-0.r20.miamfl02.us .??????......???????......??????.......??????.....>.??????..
12. as-2.r21.asbnva01.us.bb. .??????......???????......??????.......??????.?...>.??????.
13. po-4.r05.asbnva01.us.bb. ???????.?.????????????...??????????..????????.???.?????????
14. 64.208.110.253           ???????......???????......??????.......??????..?...???????.
15. 208.178.61.66            ???????......???????......??????.......??????......???????.
16. vlan1455-10ge.c1.hmg.osl ???????......??????.......??????.......??????....>.???????.
17. c1.hmg.osl.no.webdealnet ???????......??????.......??????.......??????......???????.
18. vuizook.err.no           ??????.......??????.......??????.......??????......??????..
Scale:  .:41 ms  1:101 ms  2:161 ms  3:301 ms  a:661 ms  b:1002 ms  c:1602 ms
This is my current internet connectivity. Yay, or something.

24 November 2008

Tollef Fog Heen: How to handle reference material in a VCS?

I tend to have a bunch of reference material stored in my home directory. Everything from RFCs, which is trivial to get at again using a quick rsync command (but immensely useful when I want to look up something and am not online) to requirements specifications for systems I made years and years ago. If I didn't use a VCS, I would just store those in a directory off my home directory, to be perused whenever I felt the need. Now, with a VCS controlling my ~, it feels like I should be able to get rid of those, and just look them up again if I ever need them. However, this poses a few problems, such as "how do you keep track of not only what is in a VCS, but also what has ever been there". Tools like grep doesn't work so well across time, even though git has a grep command too, it still doesn't cut it for non-ASCII data formats. Does anybody have a good solution to this problem? I can't think I'm the only one who have the need for good tools here.

26 September 2008

Tollef Fog Heen: Things to do on a Friday night

(when your wife/SO is not around) Hurrah! And it only took like six or seven hours. Notes for later: This is an YDL Powerstation, the magic mkofboot invocation is mkofboot -b /dev/sda1 --filesystem raw; the raw bit is needed for IBM hardware. The firmware also cares about whether the partition is marked as bootable or not, so make sure to mark it as such.

22 September 2008

Tollef Fog Heen: Collaborative editing, Emacs and D-Bus

A little while ago, I discovered Emacs has D-Bus support now. I wanted to play around with it, and I discovered Alban's collaborative editing page. It seems like Emacs's D-Bus documentation is somewhat lacking, though CVS has some more docs. Alban's work seems to need a well-known name, something Emacs can't yet do, so I'll probably add a method to text-remote-control where any app can register and say "Hi, I support your interface" as I think this makes more sense. I don't have any code that does anything useful yet, but once text-remote-control is fixed, I don't think that's very hard.

2 September 2008

Tollef Fog Heen: eweouz 0.2 released

I finally got around to releasing eweouz 0.2 tonight. It is still a bridge between emacs and Evolution Data Server, now with both gnus and wanderlust support. Changes include: eweouz now also has a mailing list and a git and gitweb repositories.

8 August 2008

Tollef Fog Heen: eweouz 0.1 released

I just released the first version of eweouz, the interface between emacs and evolution-data-server. Sources are available, no packages yet. It is now possible to both add contacts from within gnus and search for contacts. Patches and feedback is of course welcome.

22 July 2008

Tollef Fog Heen: Kernel patches and the TEMPer USB thermometer

Today, I submitted my two first patches to the kernel. One is fairly trivial (remove a USB ID from the pl2303 driver), the other is a fixup of a patch from some other guy, which adds support for DTR, RTS and CTS to the ch341 driver. Yay, hopefully they'll get accepted. The reason for this journey into kernel land is I got a TEMPer USB thermometer in the mail yesterday. After a fair bit of digging around, I found some source code to drive it from C# and Windows. I wanted to poke at it from Linux and C. After more poking, I found it's an I2C device connected to the USB serial adapter, where you talk to it by twiddling DTR, RTS and CTS. The C# code was fairly easy to port, so now I can get temperature readings from the command line. The code is a quick hack, but I guess it might be interesting to some people. Oh, and the calibration seems entirely off (but it was off in Windows too), so feedback on whether it's more accurate for other people would be appreciated.

14 July 2008

Tollef Fog Heen: eweouz (bbdb-a-like) working

Some years ago, when I hacked on a TDB backend for Evolution Data Server (EDS), I also wrote something about writing some command line tools for accessing my contacts and so on. I've finally gotten my act together and wrote the necessary glue to have addres completion between Gnus and EDS. It's nowhere as feature-complete as BBDB is, but it allows you to complete addresses, at least. Get it from git (gitweb). I'll hopefully expand it a little bit and cut a release fairly soon.

12 June 2008

Tollef Fog Heen: Zombie meme

Various people have been posting about what to do when the zombies attack.
You are in a mall when zombies attack. You have:
1. One weapon
2. One song blasting on the speakers
3. One famous person to fight along side you.
  1. Katana. Doesn't run out of ammo and should slice up those zombies quite nicely.
  2. One of the songs from the soundtrack of "The Rock". I can't remember which right now, since it seems to have disappeared from my music collection.
  3. Given I'm facing undeads, somebody holy. I suppose Jesus might work well.

19 May 2008

Tollef Fog Heen: New backup system!

(This post is mostly as a reminder to myself on how I've set up my backup system. It should probably go on a wiki instead so I can keep it up to date.) After the recent OpenSSL debacle in Debian and Ubuntu, I found that all my backups were encrypted with something amounting to a well-known secret key. Ouch. I was not entirely happy with how my old backup system worked either (it was based on boxbackup). In particular, the on-disk format was opaque, the tools needed to access it were not particularly user-friendly and I had to run Yet Another CA for managing the keys for it. After looking around a little, I settled on rdup which is a tool very much written in the unix tradition of "do one thing and do it well". As it reads on the home page:
The only backup program that doesn't make backups!
(which is almost true). It keeps a list of information about which files have been backed up locally on the machine to be backed up, including some meta-information such as file size and permissions, so it can take a new backup if any of those changes. For more details, read the web page and the source. rdup is more of a framework for making your own backup system than a complete system in its own right, so this post is really about how I have customised it. First, I want my backups to be encrypted, and rdup supports encryption (both GPG and mcrypt). I'm lazy, so I settled on what rdup-simple gives me, which is mcrypt. Key generation is easy enough: head -c 56 /dev/random > /root/backup-$(hostname).crypt.key and then a chmod 600 to avoid it being world-readable. In /root/.ssh/config, I put
Host backup-$hostname
Hostname $backupserver.err.no
User backup-$hostname
IdentityFile /root/.ssh/id_rsa_rdup
ProxyCommand pv -L 40k -q   nc %h %p
so as to make it fairly easy to move stuff around and to make it pick up the right identity. The last bit is a trick to rate limit it so it doesn't saturate my DSL. pv has a wonderful -R switch which lets me change the arguments to an already-running pv, if I want to do that. ssh-keygen -t rsa -f /root/.ssh/id_rsa_rdup to generate an ssh key. It got put into /home/backup-$hostname/.ssh/authorized_keys on the backup server, so the line reads like:
command="/usr/local/bin/rdup-ssh-wrapper",no-pty,no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-rsa AAAAB3N
The /usr/local/bin/rdup-ssh-wrapper is a small perl wrapper which only allows the rdup commands and sanitises the command line somewhat. Since I don't want to make a backup of all bits on my machines, I have an exclude file, which lives in /root/rdup-exclude. It is just a list of regexes of files to ignore. To actually make a backup, I run something like for p in /etc /home /var; do rdup-simple -v -a -z -E /root/rdup-exclude -k /root/backup-$(hostname).crypt.key $p ssh://backup-$(hostname)/srv/backup/$(hostname)/$p ; done which then goes on for a while. It gives me nice structures with hard-linked files to avoid using more disk space than needed. I can then just have a small find(1) script prunes old backups as I don't need them.

31 March 2008

Tollef Fog Heen: Changing jobs

15:58 [Canonical] -!- Irssi: Disconnecting from server irc.canonical.com: [kthxbye!]
15:58 [Canonical] -!- Irssi: Connection lost to irc.canonical.com
From tomorrow on, I'm working full-time for Linpro, a Norwegian Linux consulting company.

25 March 2008

Tollef Fog Heen: pkg-config, sonames and Requires.private

This post is both an attempt at replying to a bug against telepathy-glib, but also an attempt at explaining what Requires.private do (and don't). I am using Evolution as my example here, not to pick on Evolution or its authors in any way, but because it's a convenient example. Currently, on Ubuntu Hardy, evolution links against 75 different libraries. Amongst those, we find libz.so.1, libXinerama.so.1 and many more. I'll go out on a limb here and claim that Evolution does not call any of the functions in libXinerama directly. Let that be the assumption from here on. An obvious question then is, why does evolution link against libXinerama.so.1 if it doesn't use it? To answer that question, we need to go back in time to before we had dynamic linking. If you wanted to build a binary like evolution you had to have 75 -l statements when you linked and you ended up with the whole code for Xinerama embedded in your email and calendar client. For various reasons, we stopped doing that and switched to dynamic linking where the evolution binary just contains a reference to libXinerama. At some point we also grew the ability for libraries to contain those references to other libraries, so you don't have to hunt down all the dependencies of libfoo when you are linking with it. We also got tools such as libtool which try to abstract away a lot of the problems of building on older platforms which don't support inter-library dependencies. Now, since evolution still doesn't use anything directly in libXinerama.so.1 but just uses a library which in turn links against libXinerama.so.1, it shouldn't be linking against it. Then why is it linked with it? Again, we need to look back at history, and for this part I am at least partially responsible. pkg-config was originally written as a replacement for gnome-config and various other -config utilities. Lots of libraries and applications now ship .pc files and we have a standardised interface for querying those files. One of the problems the original authors of pkg-config faced was the problem of dependencies. They added dependencies so the authors of gst-python-0.10 could say "We need pygtk-2.0 too" and so the compilation flags needed for gst-python-0.10 would also include those for pygtk-2.0. Note that I'm using "compilation flags" loosely here, I am not just talking about CFLAGS. This did not fix the problem of inflated dependencies. Not at all. I talked with some of the Debian Release Managers back in 2004/2005 and we worked out a solution which should help us have correct, uninflated dependencies since the then-current way of handling dependencies caused big problems for migrations of packages between unstable and testing. The plan was to introduce a new field, Requires.private which would not show up unless you passed --static to the pkg-config invocation (since you need all libraries if you are linking statically). This definition of Requires.private was mostly useless since GNOME and GTK+ have a habit of including each other's headers. To make a long story short, I changed the semantics so the Cflags field from private dependencies were included even when not linking statically. A problem which pkg-config does nothing to guard against in this case is if you have libfoo.so.2 linking against libbar.so.1 and libfoo.so.2 exports some of libbar's types in its ABI (and not just as pointers, but actual structs and such). If libbar's soname is then bumped to libbar.so.2 and libfoo is rebuilt, libfoo's ABI has changed without a soname bump. This is bad and will cause problems. If your application is linked against both libfoo.so.2 and libbar.so.1, you'll still get problems since libfoo.so.2 then suddenly pulls in symbols from libbar.so.2. If you used symbol versioning, you would at least not get symbol conflicts and your application would continue to work, but you would have a spurious dependency and the package containing libbar.so.1 would be kept around until your application was recompiled. With this background, you might ask the question why we still have Requires since it is seemingly useless. For C, it is useless in all but the most special cases, just use Requires.private instead (and its sibling Libs.private). Other languages have different semantics. Some people use .pc files for other purposes such as gnome-screensaver having variables defining where themes and screensavers go. Hopefully this blog post has explained a bit about why we have Requires.private and what the difference between this and their regular counterpart is. If there's anything unclear, please do not hesitate to contact me.

20 March 2008

Tollef Fog Heen: Default sudo configuration in Ubuntu

Gunnar Wolf claims that Ubuntu ships a passwordless sudo by default. This would be an insane default configuration, so Ubuntu doesn't. What they do however is add the default user to the admin group which is allowed to use sudo. Incidentially, Debian does the same thing (except it's just for the first user, not the admin group) if you don't set a root password.

29 February 2008

Tollef Fog Heen: Achievo and sticky projects

I finally got tired of Achievo not remembering what projects I last used, so I whipped up a small greasemonkey script to do that. It just makes the selection sticky, nothing very fancy.

9 January 2008

Tollef Fog Heen: Choosing a nonce in CTR mode

I am currently working on implementing a cryptographic file system using FUSE. It is different from EncFS and similar in that it just mirrors a normal directory tree, but encrypts the contents of the files as they are read or decrypted as they are written. My use case is backups. I have some machines where I and only I have access, machines which may contain proprietary information, personal emails and so on. Of course, I want backups of those, so when the hard drives stop working, I don't lose any data. The machine(s) I am backing up to, however are not always machines where I trust all the people with physical access to not make a copy of my data. In addition, I don't want broken hard drives returned under warranty to contain unencrypted data. This use case is the reason for why I'm encrypting on read rather than on write. I have chosen to use CTR (counter) mode together with AES which should give acceptable security. One of the requirements CTR needs to work well is a nonce, typically 64 bits (for 128 bit AES) which must not ever be used twice. If you use it twice, you leak information about your plaintext, which is, for obvious reasons, bad. My current design headache is how to choose a good nonce. Ideally, I believe it should be persistent for each version of the file and unique per file. Using the inode number takes up 64 bits (on AMD64 at any take, or when using -D_FILE_OFFSET_BITS=64 on 32 bit platforms). So while this gives me the latter, it doesn't give me the former at all. I am wondering if I should use the inode number modulo 2^32 (effectively choosing the lower 32 bits of the inode number) and then something which is fairly sure to never be the same, such as mtime (or at least the lower 32 bits of it, when time_t becomes 64 bit). The reason for not just choosing a completely random value is I don't want a command like diff file1 file1 to claim there are differences in the file. My hope was I'd get a great idea on how to solve the problem as part of writing it down. Alas, that hasn't happened, so if you happen to come across a great solution (or a reason to avoid a particular choice), feel free to email me

25 November 2007

Tollef Fog Heen: Button pushes you!

In Soviet Boston, button pushes you. Here is proof: Button pushes you

22 November 2007

Tollef Fog Heen: Renewing CA certificates

I'm in the process of moving servers at the moment, and amongst the services I am moving is my authenticated, but public SMTP relay service. That service includes a basic web-based CA whose CA certificate is going to expire in July 2008. Apparently, it is possible to refresh CA certificates without having to distribute fresh certificates to all the clients. The magic steps, using openssl are: First, generate a new CSR using the existing certificate and key as input:
openssl x509 -x509toreq -in cacert.crt \
  -signkey cakey.key -out renew.pem
Then sign it:
openssl x509 \
  -extfile openssl.cnf \
  -extensions v3_ca \
  -CA cacert.crt -CAkey cakey.key \
  -set_serial 0 -days 365 \
  -req -in renew.pem -out newcacert.pem
This information was quite hard to come by, so hereby put here so I can find it if I need it again.

Antti-Juhani Kaijanaho: Planet Debian RSS 2.0 feed is broken

Tollef Fog Heen asks:
Also, why does bloglines link to the completely wrong place on dburrow’s posts? It links to http://planet.debian.org/tag:blogger.com,1999:blog-$blah rather than the real URL.
I think I know why. I don’t know which Planet feed Tollef is reading, but a look at the Planet Debian RSS 2.0 feed is illuminating. Looking up Daniel’s entry, we can find the following sub-element:
<guid>tag:blogger.com,1999:blog-12729767.post-4989856010680867140</guid>
Looking at the RSS 2.0 specification, we can find the following note:
If the guid element has an attribute named “isPermaLink” with a value of true, the reader may assume that it is a permalink to the item, that is, a url that can be opened in a Web browser, that points to the full item described by the element. An example: <guid isPermaLink=”true”>http://inessential.com/2002/09/01.php#a2</guid> isPermaLink is optional, its default value is true. If its value is false, the guid may not be assumed to be a url, or a url to anything in particular.
Thus the Planet RSS 2.0 feed claims that tag:blogger.com,1999:blog-12729767.post-4989856010680867140 is a valid URL. There is no basis for this claim in Daniel’s Atom feed, as that string only occurs inside an id element, and RFC 4287 specifies that the string cannot be assumed to be dereferencable.

Next.

Previous.